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IBSTBACT 

National Evaluation Systems, Inc., has developed 
curriculum evaluation techniques, in terms of learner verification, 
which may be used to help the curriculum-development efforts of 
publishing companies, state education departments, and universities. 
This document includes a summary of the learner-verification 
approach, with data collected about a new mathematics series, and a 
report about the application of the procedure to the "Series r" 
reading program developed by the Hacmillan Publishing Company. The 
report describes five components of learner verification (overall 
student achievement, specific student achievement, specific student 
growth and retention, and student and teacher opinion and 
satisfaction data) , discusses the evaluative findings for each of 
these components, summarizes the overall evaluation, and contains 
charts illustrating the findings. (JM) 



^^^(^^^(♦♦♦♦♦♦♦♦^'♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦^♦♦♦♦♦♦♦^♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦^ 

♦ Documents acquired by EBIC include many informal unpublished ♦ 

♦ materials not available from other sources. EBIC makes every effort ♦ 

♦ to obtain the best copy available. Nevertheless, items of marginal ♦ 

♦ reproducibility are often encountered and this affects the quality ♦ 

♦ of the microfiche and hardcopy reproductions EBIC makes available ♦ 

♦ via the ^IC Document Beproduction Service (EDBS) . EDBS is not ♦ 

♦ respa le for the quality of the original document. Beproductlons ♦ 

♦ suppli d by EDBS are the best that can be made from the original. ♦ 
♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 



EKLC 



us DEPARTMENT OF HEALTH. 
EDUCATION AWELFARE 
NATIONAL INSTITUTE OF 
EDUCATION 

TM.S DOCUMENT MA'. BFFN fttPRQ- 
OUCEO EXACTLY AS oeCEiVtO »^ «OM 
THE PERSON OR ORGANIZATION ORJ&tN- 
ATING»T POINTS O*^ VIEW OR 0P»NI0NS 
STATED DO NOT NECESSARILY REPRE- 
SENT OF F »ClAL NATION AL INSTITUTE OF 
EDUCATION POSITION OR POLICY 




Dear R a 




aire 





ERIC 



THE LEARNER VERIFICATION OF 
SERIES r: TEE IBW ^'lACmLIAU HEADIuG FKOGFJh'-I 



HIGHLIGHTS 



Prepared by: 



National Evaluation Systems, Inc. 
Post Office Box 226 
Amherst, Massachusetts 01002 



William Phillip Gorth, Ph.D., President 
Peter E. Schriber, Ed.D., Vice-President 
Richard G. Allan, Ed.D., Vice-President 
Janet C. Ziegenhagen, Project Coordinator 



July 15, 1975 



The letter that appears on the cover of this report 
submitted by a student in St. Louis, Missouri. 



INTRODUCTION 

Throughout the school year 1974-75, National Evaluation Systems, Inc., 
conducted a learner verification of Series r: Tne Uew Maomillan Reading 
Progrccn. This surmnary report contains highlights from the information that 
was collected during the term of the verification. Five components of the 
learner verification are described below. 

Component A: MeasureTOnt of Overall 
Student Achiever^ent Usino Ser^ies v 

Component A employed a number of widely known, norm-referenced 
achievement tests to measure student growth during the period between the 
administration of a Fall pretest and a Spring posttest. These commercially 
available tests provide highly credible measures of student growth with 
which teachers, administrators, school boards, and state departments of 
education can evaluate the effectiveness of :^er-ies r. 

Component B: Specific Student Achiev ement 
at Four Levels of jcviez v 

Component B included the administration and analysis of Series r 
Post-Assessment Tests at Levels 6, 11, 22 and 28. Through the analysis of 
a large stude . ^.mple's test results. National Evaluation Systems was able 
to determine the degree of student mastery of specific focus skills at 
selected levels. 
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Component C: Specific Student Growth and 
Retention Across Levels of Ssries r 

Component C included two sets of parallel test forms developed by 
National Evaluation Systems for use at the third- and fourth-grade levels. 
The objective of this component was to determine, with an independently 
developed instrument, student pre-, post-, and retention performance on 
approximately 10 of the focus skills at each of the two grade levels. 

Components D and I: Student and Teacher 
Opinion and Satisfaction Data 

Components D and E involved the use of four types of questionnaires 
in both the Fall and Spring. Opinion and satisfaction data for both sam- 
plings were gathered through the use of a "Teacher Opinion Questionnaire," 
"Student" Opinion Questionnaire for Kindergarten through Grade 3," "Student 
Opinion Questionnaire for Grades 4 through 6," and a "Field Test Monitor 
Questionnaire." The administration of questionnaires enabled National 
Evaluation Systems to collect opinions and suggestions covering a wide 
range of topics from approximately 2,500 students and over 100 teachers 
in the Fall and again in the Spring. 

The learner verification of Series r involved 173 classrooms across 
six regions of the country: Region 1 (Northeast), Region 2 ^Southeast), 
Region 3 (Southwest), Region 4 (Midwest), Region 4A (Great Lakes), and 
Region 5 (West). Table 1 on the following page contains a pilot classroom 
distribution chart by region, grade level, and community type. 
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Throughout the school year each of the 173 participating pilot teach- 
ers, approximately 4,500 students in grades K-5, and 36 regional Macmillan 
sales representatives participated in the learner verification. 

« 

SUMiMARY OF RESULTS FOR CC"PONENT A: 
MEASUREMENT OF OVERALL STUDENT ACHIEVEMENT 

The primary purpose of measuring student achievement with established 
andv/idely accepted tests of basic verbal skill was to determine how well 
students using SsHes r reading materials were learning reading skills and 
achieving in related skill areas. A secondary purpose was to identify any 
patterns of student performance within and across grade levels which showed 
differing reading grov/th rates for community types and for particular sub- 
tests of the testing batteries. 

Below are suir^i^aries of student growth between the Fall and Spring test 
administrations. 

Grade 1 . These students were included in the. Spring administration 
only. For all four of the verbal subtests of the Stanford Early School 
- Achievement Test and the three subtests of the Cooperative Primary Test, 
the results were positive. Students achieved at or above average perfor- 
mance on all seven subtests (for example: Sentence Reading, sixty-second 
percentile; Reading, sixty-eighth percentile; Word Analysis, seventy- fourth 
percentile). 

Grade 2 . Highlights included good achievement on the Cooperative 
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Primary Test, with special notice of the Word Analysis subtest, on which 
the group achieved a percentile rank of 65. On the Iowa Tests of Basic 
Skills, students were expected to gain one month in grade equivalent score 
for each month between the Fall and Spring administrations. Excellent per- 
formance was indicated by the Total Language score, which showed that the 
students exceeded expected growth by one month. 

Grade 3 . The results of the Comprehensive Tests of Basic Skills 
showed the third graders to have exceeded the expected grade equivalent 
scores in the Spring on all six subtests. In five out of six cases the 
amount by which the expected scores were exceeded was between three and 
f.ive months* growth. All subtests showed grade equivalents of 4.0 years 
or higher. 

Grade 4 . On both the Comprehensive Tests of Basic Skills and the Iowa 
Tests of- Basic Skills, the fourth graders as a group equalled or exceeded 
the expected grade equivalent scores for the Spring test administration. 

Grade 5 . On both the Comprehensive Tests of Basic Skills and the Iowa 

Tests of Basic Skills, the fifth graders as a group equalled or exceeded 

the expected grade equivalent scores for the Spring test administration on 
15 of the 16 subtests. 

Grade 6 . On six of the seven verbal subtests of the Comprehensive 
Tests of Basic Skills, the sixth graders as a group achieved at or above 
the expected grade equivalent scores for the Spring administration. Of 
particular note was student performance for Total Study Skills, which 
exceeded expected growth by nearly seven months. 
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Across the six grade levels the overwhelming result is that in all but 
a very few instances the student groups at each grade level achieved af or 
above the expected grade equivalent scores. There were frequent instances 
in which perforn^.ance exceeded expectation by several months. 

An interesting pattern is that students at different grade levels 
exceeded expected performance on different subtests. This is one indicator 
of the ability of Sevies v to cover a wide range of language arts skills in 
addition to reading com.prehension . 

There were no identifiable negative patterns across grade levels for 

specific subtests. 

An overview of the results by community type indicates that regardless 
of community type (inner-city, suburban, rural), students tended to achieve 
at or above the expected growth represented by the grade equivalent scores. 

Reading Figures 1, 2, 3, 4 and 5 . For the Comprehensive Tests of 
Basic Skills and the Iowa Tests of Basic Skills, scores were converted to 
grade equivalent scores. For example, a grade equivalent score of 4.5 in- 
dicates that the student performed on a specific test as a student midway 
through the fourth grade (i.e.-, fourth grade, fifth month) would be expected 
to perform on that test. Thus, if a group's average grade equivalent score 
in the Fall is 4.1 and the group is administered the Spring test six months 
later, the expected achievement would be one month's growth for each month 
of school, or a Spring grade equivalent score of 4.7 (4.1 + 0.6 = 4.7). 
Figures 1, 2, 3, 4 and 5 highlight results of Component A. 
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SUI-'uMARY OF RESULTS FOR CCMPGNEMT B: SPECIFIC 

STUDENT ACHIEVEME;^ AT FOUR LEVELS OF SERIES r 

« 

This cor.ponent of the verification provides evidence to confirm student 
learning by determining whether students demonstrate competency in specific 
focus skills at various levels through the analysis of the Mscmillan Sene- 
r Post-Assessment Test results. Thus, given the student assessment tests 
developed by the authors of the Series r program, a student passing through 
each level should show achievement at the conclusion of instruction for a 
level. 

For all four reading levels examined in this component, the students 
clearly showed mastery of the focus skills on the author-constructed Post- 
Assessment Tests. 



SUMMARY OF RESULTS FOR COr'POriENT C: SPECIFIC STUDEHT 
GROWTH AND RETENTION ACROSS LEVELS OF SERIES r 

The Hacmillan ScHcc r program has many materials designed to insure 
Student growth and retention, which can be confirmed by administering tests 
Specifically designed to document growth and retention across levels of 
SeHes r. The methodology used for this component of the verification 
Offers an example of what can be done to try out materials for any number 
of Series v levels over a period of tine with student groups of any size. 
The objective of this component was to determine students' ability to 
master and retain skills defined as "focus skills" t.;c grade levels 
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of the Scin.es v program. 

It is evident from the results of pre-, post-, and retention testing 
of two selected Sevzca r levels for grades 3 and 4 that students tend to 
retain skills once learned. 

A second conclusion points to the usefulness of having longitudinal 
information on student perfor::iance--i .e. , data which show learning trends 
on a set of focus skills over time. These data are useful for: 

1) identifying student readiness for successive Series r levels; 

2) identifying focus skills which are unusually easy or difficult 
for large numbers of students before, after, and long after 
instruction; 

3) determining which focus skills build best from one Series r 
level to another Series r level; 

4) determining which focus skills are reinforced most subsequent 
Series r levels; and 

5) determining which focus skills need to be built upon or reinforced 
by subsequent levels. 

SUW^ARY RESULTS FROM COMPONENTS D AND E: 
STUDENT OPINION DATA AND TEACHER SATISFACTION DATA 

The ultimate consumer of Macmillan's Series v materials is the student. 
If students enjoy the materials, show extended interest, and demonstrate 
substantial progress in acquiring reading skills, Sr-ics r will be 
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successful. Therefore, student opinions are as vital to learnsr verifi- 
cation as are teacher opinions. The first goal of this component was to 
gather student opinions and attitudes about the Ser^iee r reading program. 
The second goal was to gather inforration on teacher opinions and attitudes. 
The third goal was to determine how teachers are instituting the Series r 
program in the classroom by personal interview and classroom observation. 

The following is a brief summary of the major findings of the opinion 
queries including mail questionnaires, interviews, and observations. 

Students : 

1) high interest in story content and artwork of students of all 
reading levels and grade levels (across grade levels at least 
two-thirds of all students answered that they liked alj_ the 
stories) ; 

2) -high appeal of the workbook to many students (by Spring over 

80% of the students at each of the six grade levels reported 
that they liked the v/orkbook); 

3) general lack of stories of primary interest to only boys or 
only girls. 

Teachers; , . 

1) satisfaction that students are enthusiastic about reading 

material s; 

2) ease of lesson preparation; 

3) impetus toward grouping for or individualizing reading along 
with a built-in management structure; 

17 
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4) v/ell-liked and often-used Teacher's Edition. 

Highlights of the student and teacher responses are shown in Figures 
6, 7 and 8. 

SUM.'-'ARY OF ALL COMPONENTS 

4 

The vast majority (over 90."^ of teachers and students taking part in 
the learner verification heartily endorsed Series r and enjoyed using the 
series. Results of the standardized test administrations in the Fall and 
Spring show that, at the very least, students tended to gain in language 
arts and reading skills at the expected rate in most subtest areas. In a 
wide variety of areas, depending on grade levels, students achieved above 
the expected rate. 

The learner verification student samples represented a wide geograph- 
ical range as well as a good mix of inner-city, suburban, and rural areas. 
The results clearly show that Series r was well liked by students and 
teachers of all areas and that student growth in reading and language arts 
was generally very good, regardless of geographic location or community 
type. 
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ABSTRACT 

A Successful Approach to Curriculum 
Evaluation/Learner Verification 

RICHARD G, ALLAN, National Evaluation Systems, Inc, 
WILLIAM P, GORTH, National Evaluation Systems, Inc, 
PAUL D, PINSKY, National Evaluation Systems, Inc. 

Curriculum evaluation in terms of learner verification is an increas- 
ingly interesting issue due to economic, legislative and marketplace pres- 
sures. National Evaluation Systems, Inc, , has been in the forefront in 
applying existing technology to major curriculum development efforts and 
creating or applying new technological techniques to curriculum evaluation, 

NES has developed curriculum evaluation techniques to support the 
development of an improved and marketable product. The evaluation does not 
interfere v/ith delicate production schedules and has proven cost-effective 
for publishing companies (Macmillan, Harper and Row), state education 
departments (New York), and university-based curriculum developers (Social 
Studies Curriculum Development Center, University of Indiana), 
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Curriculum evaluation in terms of learner verification is an increasingly 
interesting issue due to economic, legislaqive and marketplace pressures. 
National Evaluation Systems, Inc. (NES) has been in the forefront in applying 
existing technology to major curriculum development efforts and creating or 
applying new technological techniques to curriculum evaluation. 

NES has employed standardized tests, curriculum-based criterion-referenced 
tests, NES-produced criterion-referenced tests, teacher and student question- 
naires, observation techniques, and interviev/ forms to gather appropriate data. 
The data collected have included mastery, comparative, longitudinal, opinion, 
and achievement data. The achievement data have been provided for many subsets 
of the population, such as (1) region of the county, (2) type of school dis- 
trict, and (3) type of classroom organization. 

An interesting aspect of the NES studies is that they were not conducted 
for well-financed experimental programs, but for publishing companies, state 
education departments, and university-based curriculum development efforts, 
where cost efficiency and benefit for the dollar are prime considerations. 
Curriculum development project directors agree the above has been achieved. 
We have conducted curriculum evaluations for basal reading and basal math 
programs, television programs, and others for companies/agencies such the 
Macmillan Publishing Company, Harper and Row, Publishers, the New York State 
Education Department, and an NSF-funded project at the Social Studies Curric- 
ulum Development Center at the University of Indiana. 

a) Objectives : Given a curriculum program, NES will: 

1) identify information useful to the curriculum developers; 

2) collect the information; 

3) report the information in a timely and usable manner. 

b) Perspective : Curriculum evaluation is quite often given lip service only. 
Major problems in the past have been the time required for conducting a 
curriculum evaluation and the cost benefit of such a study. We have been 
able to provide tinely information for both product development and 
marketing purposes at a level deemed cost efficient by publishing companies 
and curriculum developers. We feel the techniques we are using for major 
curriculum evaluations represent a step forward for the general curriculum 
developer as sophisticated information is supplied with little delay to the 
curriculum-development process at an affordable cost. 
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c) Methodology : A systematic approach to collecting evaluation data is used. 

1) Identify the instructional materials for learner verification. 

2) Decide upon scope of learner verification. 

3) Allocate resources for conducting the learner verification. 

4) Select test sites. 

5) Set up data collection schedule. 
5) Prepare objectives and items. 

7) Produce tests. 

8) Produce questionnaires, checklists, interviews, 

9) Collect data. 

10) Score, process, and tabulate the data. 

11) Analyze the data. 

12) Prepare learner verification reports. 

d) Data source : The data sources are drawn from among (1) students, (2) 
teachers, (3) educational specialists, and (4) content specialists, 

e) Results and conclusions : The results indicate that curriculum evaluation 
is a viable process that can contribute to an improved and marketable 
product for curriculum development projects. Also, the evaluation does 
not interfere with the developmental timetable or be so costly as to be 
prohibi ti ve. 

f) Educational importance of the study : We believe our experience in applying 
existing curriculum evaluation techniques and in developing new curriculum 
development techniques has proved the feasibility of curriculum evaluation/ 
learner verification if appropriately applied. Our experience has shov/n 
that efforts to comply with new state laws (California, Texas, Florida, 
etc.) rather than resistance to the laws (see comments appended to EPIE 
LVR report) is a sensible and productive avenue of pursuit. 

Some of the more interesting data we produce, not usually available in a 
major curriculum evaluation, are shov/r below. This study was conducted for a 
major New York publisher for a new basal mathematics series. The test was con- 
ducted on 1,500 students to collect curriculum development data in 1973, A 
follow-up study is currently under way to collect data nationwide on a larger 
sample of students. 

The data represent three units at the third-grade level. All data are 
longitudinal (i.e., objectives for all three units were tested at each of four 
points in time). Chart 1 shows the success of various groupings of students 
before instruction (PRE), after instruction on Unit 1 (Ul), after instruction 
on Unit 2 (U2), and after instruction on Unit 3 (U3). Chart 2 shows three 
levels of breakdown for all objectives in all three units, the objectives in 
each of the three units, and selected objectives within each of the units. 
Chart 3 shows a comparison between results on the curriculum-based END OF UNIT 
TEST and the more sensitive longitudinal data. 
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Math-3 Units (3rd Grade) 



PRE 



U1 



U2 



U3 













All ^tuHpnt*; on All Q\ 


31 


53 


58 


75 


llrhan ^tnHpnt*; On All 0\ 


26 


50 


56 


66 


Suburban Students on All O's 


36 


54 


66 


80 


Rural Students on AM O's 


30 


54 


67 


76 


Inner-City Students on AH O's 


13 


35 


54 


68 


Slow 


21 


51 


58 


67 


Average 


33 


54 


64 


77 


Fast 


47 


65 


70 


78 


Heterogeneous 


29 


50 


62 


79 



CHART 1 



CHART 2 



Math-3 Units (3rd Grade) 





PRE 


LM 


U2 


U3 


All O's 


31 


53 


58 


75 


Unit 1 Numeration 


28 


71 


72 


74 


Unit 2 Addition 


36 


52 


76 


76 


Unit 3 Subtraction 


22 


29 


37 


70 


Objective 101 


12 


80 


80 


83 


Objective 102 


1 


48 


48 


50 


Objective 108 


35 


60 


59 


65 


Objective 105 


36 


86 


86 


89 


Objective 201 


46 


56 


86 


85 


Objective 202 


25 


34 


71 


67 


Objective 203 


49 


61 


86 


87 


Objective 204 


21 


25 


63 


66 


Objective 303 


32 


43 


55 


85 


Objective 305 


22 


24 


31 


63 


Objective 307 


4 


5 


9 


57 



ERIC 







Math- 


-3 Units (3rd Grade) 


OBJ 


PRE 


U1 


U2 


U3 


END OF UNIT TEST 


1U4 


52 


89 


93 


93 


91 




35 


91 


90 


90 


91 


106 


62 


92 


89 


93 


96 


109 


25 


44 


65 


72 


93 




43 


94 


84 


86 


•37 


202 


37 


59 


89 


76 


93 


302 


16 


16 


20 


70 


93 




12 


19 


17 


75 


94 
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